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1 Introduction 

The main purpose of this paper is to prove the existence of fixed points for some collections 
of holomorphic maps of the unit ball B in the space L{K, H) of operators from a finite- 
dimensional Hilbert space to a Hilbert space H . More precisely we consider 1) groups 
of biholomorphic maps, and 2) finite commutative families of holomorphic maps, and prove 
that in both cases the existence of a fixed point is equivalent to the existence of an invariant 
set separated from the boundary of B. For the first case, this result was obtained in |17j : 
the advantage of the present approach is that we do not use the deep theory of Shafrir [23] 
replacing it by some simple general observations from metric geometry. 

The standard tool in the study of holomorphic maps of B is the Caratheodory metric 
p = cb'- it is known that holomorphic maps do not increase cg. A similar metric p can 
be defined in the case where Hilbert spaces K and H are over reals. We treat both cases 
simultaneously dealing with p-non-expansive maps. This allows us to obtain applications 
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to invariant subspaces and the orthogonalization problem in real Hilbert spaces similar to 
those that were obtained in [T7] for the complex case. Namely it is proved that a bounded 
group of J-unitary operators in a real Pontryagin space has a maximal negative invariant 
subspace and that a bounded representation of a group in a real Hilbert space that preserves 
a sesquilinear form with a finite number of negative squares is similar to an orthogonal 
representation. We also remove the assumption of separability of H made in [IT| . 

Our main technical tool is the proof of a kind of normal structure in E - the existence 
of non-diametral points for a sufficiently rich class of sets. In more detail: first we establish 
that each pair X,Y & B has at least one midpoint, that is, a point Z & B satisfying 
W) < l/2(p(X, W) + p(Y", W)) for all W e B. We call a subset of B mzd-convex if with 
any two points it contains all of their midpoints. WOT-compact mid-convex subsets of B 
are called m-sets. Then we prove that each m-set either is a singleton or has a non-diametral 
point (Theorem 13. 8p . 

The next step is to show that the presence of a p-bounded orbit implies the existence of 
an invariant m-set. After that we use WOT-compactness and get the existence of a minimal 
invariant m-set. Then we apply Theorem 13.81 to prove that such minimal set should be a 
singleton. 

The described general scheme for proving fixed point theorems for nonexpansive mappings 
goes back to This scheme was gradually developed by many different authors, and a 
large variety of realizations of it was found in further work on fixed points of nonexpansive 
mappings, see [6], [11], [12], [20], [21], numerous examples of applications of this scheme can 
be found in fl3\. 

Note that another instance of the connection between boundedness of orbits and the 
existence of fixed points can be found in [2U] . 

2 Metric geometry: the midpoint property, fixed po- 
ints of groups isometries, and other tools 

Let {X, d) be a metric space. By a hall in (A*, d) we mean (unless it is explicitly stated 
otherwise) a closed ball Ea^r = {x G X : d{a,x) < r}. We say that {X.,d) is hall-compact 
if a family of balls has non-void intersection provided each its finite subfamily has non-void 
intersection (see [10]). It is easy to show that each ball-compact metric space is complete. 

A subset M C A" is called hall-convex if it is an intersection of a family of balls. It is 
clear from the definition that each ball-convex set is bounded and closed. The compactness 
extends from balls to all ball-convex sets: 

Lemma 2.1 ([lO]). Let (A", d) he hall-compact. A family {M\ : A G A} o/ hall-convex suhsets 
of X has non-empty intersection if each its finite suhfamily has non-empty intersection. 

Proof. For each A G A, we have Mx = Hi^i^E^ where all are balls. Let us consider 
the family of balls U = {E^ : A G A, z G /a}. Each finite subfamily of U has a non-void 
intersection because it contains the intersection of corresponding M\. By ball-compactness, 
U has non-zero intersection which clearly coincides with r\\M\. □ 
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The diameter of a subset M C X is defined by 

diamM = sup{d{x, y) : x,y e M}. (2.1) 

A point a in a bounded subset M is called diametral if 

snp{d{a,x) : x G M} = dianiM. 

A metric space X is said to have normal structure if every ball-convex subset of X with more 
than one element has a non-diametral point. A mapping g : X ^ X is called nonexpansive 
if d{g{x), g{y)) < d{x,y) for all x,y & X. 

The concept of normal structure, introduced by Brodskii and Milman for Banach 
spaces, has played a prominent role in fixed point theory for nonexpansive mappings. See, 
for example, [6], [H], [I3], [21], [22]. In particular the scheme of the proof of the following 
result is well known; nevertheless the result itself seems to be new. 

Theorem 2.2. Suppose that a metric space {X, d) is hall-compact and has normal structure. 
If a group of isometrics of {X , d) has a hounded orhit, then it has a fixed point. 

Proof. Let G be a group of isometrics of {X , d) and let G{x) be a bounded orbit, where x is 
some point in X . Then the family $ of all balls containing G{x) is non-empty. Since G{x) 
is invariant under G, the family $ is also invariant: g{E) G for each E' G $. Hence the 
intersection Mi of all elements of $ is a non-void G-invariant ball-convex set. 

Thus the family M. of all non-void G-invariant ball-convex subsets of X is non-empty. It 
follows from Lemma 12.11 that the intersection of a decreasing chain of sets in M. belongs to 
M.. By Zorn Lemma, M. has minimal elements. Our aim is to prove that a minimal element 
M of consists of one point. 

Assume the contrary and let diamM = a > 0. Since (A', d) has normal structure, M 
contains a non-diametral point a. It follows that M (Z {x E X : d{a, x) < 6} for some 
6 < a. Set 

O = fl E,^s- 

b&M 

The set O is non-empty because a & O. Furthermore O is ball-convex by definition. To see 
that O is a proper subset of M take b,c E M with d{b, c) > 6, then c ^ Eb^s, hence c ^ O. 

Since G is a group of isometric transformations and M is invariant under each element 
of G, the action of G on M is by isometric bijections. Therefore O is G-invariant. We get a 
contradiction with the minimality of M. □ 

The following definition is essential for our results on normal structure and non-diametral 
points. 

Definition 2.3. A metric space {X, d) is said to have the midpoint property if for any two 
points a,b E X there is c G A* such that 

d{c,x) < {d{a,x) + d{b,x))/2 V x G {X,d). 

Each point c satisfying this condition is called a midpoint for (a, b), the set of all midpoints 
for (a, b) is denoted by m(a, b). 
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Remark 2.4. A straightforward adaptation of the well-known argument of Menger [16] (see 
also ^ Theorem 14.1]) shows that complete metric spaces with the midpoint property have 
the convex structure defined by Takahashi but we do not need this fact. 

Our proof of the fact that operator balls have the midpoint property uses the following 
definition and lemma. We say that an isometric involution a of A* is a reflection in a point 
xo E X a d{x, = 2d{x, xq) for each x E X. 

Lemma 2.5. // there exists a reflection a in a point Xq E X , then Xq is a midpoint for each 
pair {x, (y{x)). 

Proof. Let x E X. For each y E X, d{y,Xo) = ^d{y,(r{y)) < ^{d{y,x) + d{a{y),x))) = 
^{d{y, x) + d{y, a{x))). This is what we need. □ 

Our next result is the main technical tool for proving results on normal structure for 
spaces with the midpoint property. 

Lemma 2.6. Suppose that a metric space {X, d) has the midpoint property. Let M be a 
ball-convex subset of X and a be the diameter of M. If all points of M are diametral, then 
M contains a net {cx : A G A} with the property: limxd{cx,x) = a for each x E M. 

Proof. For ai,...,a„ E X, we denote by m(ai,...,a„) the set of all points a E X satisfying 
d{a,x) < {d{ai,x) + ... + d{an,x))/n for each x E X. The easy induction argument shows 
that if X has the midpoint property and n = 2^, then m(ai, a„) is nonempty. 

Let F = (oi, an) be a finite subset of M and let e > 0. We will show that there is a 
point c = c{F, e) such that d{ai, c) > {1 — e)a for all i. It is easy to see that a set with the 
midpoint property is either a singleton or it is infinite. Adding new points we may assume 
that n = 2^. Let b E m{ai, a„). If F C Ea^r then d{b, a) < {d{ai, a) + ... + d{an, a))/n < r 
so b E Ea^r- Thus b belongs to each ball that contains M; by ball-convexity, b E M. 

Since b is diametral, there is c G M with d{b, c) > (1 — -)a. It follows that (1 — -)a < 
n Sfc=i ^('^fc, c). If d{aj, c) < (1 — e)a, for some j < n, then 

1 , 1 , n — 1 e 

— > d[aj, c) < —{I — e)a H a = (1 )a, 

n ^-^ n n n 

k=l 

a contradiction. Hence d{aj, c) > (1 — e)a for j < n. 

Let now A be the set of all pairs {F,e) with the order given by: {Fi,ei) -< {F2,e2) if 
-^1 C F2, El > 62- For A = (F, e) E A, we write cx = c{F, e). It is clear that lim^ d{cx, x) = a 
for each x E M. □ 

The class of ball-convex sets is not sufficiently rich for our needs, because it is not closed 
with respect to unions of increasing families of sets. For this reason we introduce the following 
class of sets, which does not have this drawback. 



Definition 2.7. A subset M of a metric space is called mid-convex if m{x,y) C M for each 
pair (x, y) of points in M. 
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Clearly each ball is mid-convex. Since the class of all mid-convex sets is closed under 
intersection, each ball-convex set is mid-convex. It is also clear that each mid-convex subset 
M of a metric space with the midpoint property is itself a metric space with the midpoint 
property. This shows that Lemma 12.61 admits the following extension: 

Lemma 2.8. Let M be a bounded mid-convex subset of a metric space {X, d) having the 
midpoint property. If all points of M are diametral, then M contains a net {c\ : A G A} with 
the property: limx d{c\, x) = diamM for each a; G M. 

3 Fixed points for groups of biholomorphic maps on 
the operator ball 

Let K, H be real or complex Hilbert spaces; by L{K, H) we denote the Banach space of all 
bounded linear operators from K to H . We denote the open unit ball of L{K., H) by B and 
call it operator ball. The main goal of this section is to show that if K and H are complex 
spaces and dimK < oo, then the operator ball possesses a metric p which is invariant with 
respect to biholomorphic transformations of B and satisfies the conditions of Theorem \2.2\ 
and therefore any group of biholomorphic automorphisms of the operator ball with a bounded 
orbit has a fixed point. 

3.1 Mobius transformations and invariant distance in the operator 
ball 

For each A ^ B, we define a transformation M4 of B {a Mobius transformation) setting 

Ma{X) = (1 - AA*)-'^/'^{A + + A*X)-\l - A*Af^ (3.1) 
(this definition is due to Potapov [19]). It can be proved that 

MX' = M.A (3.2) 

(see [7j, Theorem 2). 
We set 

piA,B)=tanh-\\\M_Am\)- (3-3) 

It is easy to see that if K, H are complex, then p coincides with the Caratheodory distance 
in B. Indeed, by [5l Theorem 4.1.8], ce(0, i?) = tanh^^(||i?||) (this holds for the unit 
ball of every Banach space). Furthermore all Ma are clearly holomorphic mappings of B, 
and the formula (13. 2p shows that the same is true for their inverses. In other words. Ma 
are biholomorphic automorphisms of B. Since cb is invariant with respect to biholomorphic 
automorphisms and M^a sends A to 0, we get: 

cb{A,B) =tanlr^ ||M_^(5)|| = p{A,B). (3.4) 
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The equality (13 ■4p shows that p is a metric on B. 



To check that the same is true in the case of real scalars, one can use the complexification. 
Indeed, setting H = H (B iH, K = K ® iK one defines maps A —>■ A from L{H) to 
L{H) (and from L{K) to L{K), from L{K,H) to L{K,H), from L{H,K) to L{H,K)) by 
A{x(Biy) = AxQiAy. These maps are isometric *-homomorphisms of algebras and modules 

whence M^) = M^{B) and p{A, B) = p{A, B). 
3.2 Midpoint property of [B^ p) 

Our next aim is to prove that the metric space (i3, p) has the midpoint property. 
Lemma 3.1. p(A, -A) = 2p(0, A). 

Proof. Let A = UT be the polar decomposition of A (so T = \A\ = (A*AY^'^). Then 

p{A, -A) = tanh"^ \\Ma{A)\\ = tanh-^ ||(1 - AA*y^/^2A{l + A*A)-\l - A*AY/^\\ 
= tanh~i \\2A{1 - A*A)-^^\l + A*A)-\l - A*AY/^\\ 

= tanh-^||[/2T(l + T2)-i|| = tanh"^ ||2T(1 + T^)-! || = || tanh-^(2T(l + T2)-1)|| 
= ||2tanh"^(T)|| = 2tanh"^ ||T|| = 2tanh"^ \\A\\ = 2p(0, A) 

(we used the identity tanh(2x) = 2tanhx/(l + tanh^x) and the fact that ||/(T)|| = /(||T||) 
if a function / is non-decreasing on the interval [0, ||T||] and T is a positive operator). □ 

Lemma 3.2. For every A^B E B, there is an isometric transformation ip of {B,p) with 
ip{A) = -ip{B). 

Proof. It suffices to show this for A = 0, because the Mobius transformations (which are 
isometrics of {B,p)) act transitively on B. Thus we have to find C,D in B such that 
Mci-D) = 0, MciD) = B. Take D = C, then we have the equation MdC) = B. 
By the above calculations, this means that f/2T(l + T^)~^ = B, where UT = C is the 
polar decomposition. If i? = VS is the polar decomposition of B, then we take U = V and 
T = tanh(|tanh~^(S)). □ 

Proposition 3.3. The space {B,p) has the midpoint property. 

Proof. Let A,BeB; we have to prove that there is C & B such that p(X, C) < |(p(X, A) + 
p{X,B)) for each X E B. By Lemma [3.21 we may assume that B = —A. The involution 
a{X) = —X is clearly isometric; it follows from Lemma EH] that it is a refiection in the point 
0. Now by Lemma [2.51 is a midpoint for the pair [A, — A). □ 
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3.3 Topologies on B and ball-compactness 

A set in B is bounded with respect to p if it is contained in some p-ball, this happens if and 
only if the set is contained in a p-ball with center 0. On the other hand, it follows from the 
definition of p that a p-ball with center coincides with the closure of a multiple rB of the 
operator ball B, for some r < 1. Therefore a set is bounded if and only if it is separated 
from the boundary of B. 

The following lemma is a special case of a more general result proved in ^ Theorem 
IV.2.2] (in the case of real scalars one can use the complexification). 

Lemma 3.4. The metric p is equivalent to the operator norm on any p-bounded set. 

Thus the topology induced by the metric p on i3 is equivalent to the norm topology. 

Another important topology on B is the weak operator topology WOT (see ^ p. 476]). 
We collect the facts about the WOT that we need in the following lemma. 

Lemma 3.5. (i) Each p-ball E^^r is convex and WOT -compact. 

(ii) If dimK < oo, then each norm-closed convex subset of B is WOT -compact. 

(iii) If dim. K < oo, then all Mobius transformations are WOT -continuous. 

Proof, (i) By definition, each p-ball is the image of a p-ball with center under a Mobius 
transform. Since 0-centered p-balls are the usual balls and since a Mobius transform is 
a fractional- linear transformation, the statement follows from the fact that images of balls 
under fractional- linear transformations are convex and WOT-compact (see, for example, [9]). 

(ii) It is easy to check that if K is finite-dimensional, then the norm topology of L{K., H) 
coincides with the strong operator topology while WOT coincides with the weak topology of 
L{K, H). Since convex norm-closed bounded subsets of a reflexive space are weakly compact, 
the result follows. 

(iii) The WOT-continuity of a more general class of maps ( "fractional-linear maps with 
compact (l,2)-entries") was noticed and used by Krein in [15]. □ 

Corollary 3.6. (i) The metric space {B,p) is ball-compact. 
(ii) Ball-convex subsets of{B,p) are WOT-compact. 

3.4 The normal structure of {B, p) 

Recall that the normal structure means the existence of non-diametral points in ball-convex 
sets. We establish the existence of non-diametral points for a wider class of subsets of B. 

Definition 3.7. A subset M of i3 is called an m-set if it is mid-convex and WOT-compact. 

It follows from the results of the previous section that the class of all m-sets contains 
the class of all ball-convex sets. Furthermore, if M is a p-bounded (= separated from the 
boundary) subset of B, then there is a smallest m-set Mq containing M (the intersection of 
all m-sets containing M). We call Mq the m-span of M. 
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Theorem 3.8. Let K he finite dimensional. Then any m-set in B either is a singleton or 
has a non- diametral point. 

Proof. Let M be an m-set of {E, p) which is not a singleton. We have to prove that M 
contains a non-diametral point. 

Assume the contrary, that is, all points in M are diametral. Let a = diamM > 0. Since 
B and therefore M has the midpoint property, we have, by Lemma 12. 8[ that there is a net 
{A\} in M such that limA p{Ax, X) = a for each X G M. 

Since M is WOT-compact, the net {Ax : A e A} contains a weakly convergent cofinal 
subnet. To simplify the notation we assume that the net {Ax : A G A} itself WOT-converges 
to some operator W. Since M is WOT-compact, we have W e M. 

Since W G M, we get 

lim p{W, Ax) = a. (3.5) 

A 

We get a contradiction by proving 

sup p{Ax, Af,) > a. (3.6) 

Since all Mobius transformations are WOT-continuous isometrics of {B, p) (see Lemma 
I3.5( iii)). we may assume without loss of generality that W = (otherwise we apply M_w). 

Let P = tanha. Then (13. 5p leads to limA \ \Ax\ \ = (3 and it suffices to show that 

sup||M^^(-Aa)|| >/?. 

Since K is finite dimensional and Ax G L{K,H), we can select a strongly convergent 
subnet in the net {A^Aa}. So we assume that ^^^a —>■ P, where P G L{K,K). It is clear 
that P > and ||P|| = f3\ 

Choose e > and fix p with || AJiv4^ — P\\ < e. For brevity, denote A*^A^ by Q. We prove 
that lim^ ||M^^(— A^) || > /3 if e > is small enough. By the definition, 

MaS-Ax) = (1 - A,A*yi\A, - Ax){l - AlAx)-\l - A^A.fl^ (3.7) 

Since A* is of finite rank, A*^Ax ^ in the norm topology. Hence limA II ^^^(—^a)!! = 
limA II^aII; where 

Ta = (1 - A,Air'i\A, - Ax){i - a;a,Y/' = a,-{i- a,a;)-'/'Ax{i - a;a,Y/'. 

It follows from the identity 

^ > (l-t)(l + (l-t)-V2) 

that the operator (1 — A^A*^)^^^"^ is a finite rank perturbation of the identity operator. Since 
Ax^Oin WOT, we obtain that ||Ta - Sx\\ ^ 0, where ^a = - ^a(1 - A^A^y/^. 
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Denote Ax{l - A*^A^y/^ by Bx- Since i?A — ^ in WOT, the sequence 
(A, - BxTiA, - Bx) - A;A, - BlBx = -A;Bx - B^A, 
tends to zero in norm topology. Furthermore, 

B*xBx = il-Qy^'AlAxil-QY/' 
tends in norm topology to (1 — Qy^'^P{l — QY^"^- Therefore 

{A^ - BxYiA, -Bx)^Q + il- QY''P{1 - QY"- 
Since \\P — Q\\ < e, we have that 

HQ + (1 - QY^'Pii - QY" - (Q + (1 - Q)Q)\\ < e. 

The inequalities 

P'-e<\\Q\\<P' 

imply 

\\Q + {l-Q)Q\\>2p^-P'-2e, 

whence 

hm II^ISaII = lim \\{A^ - BxY{A^ - Bx)\\ > 2/3^ - - ?>e > I3\ 

A A 

the last inequality is satisfied if e is sufficiently small. □ 

Since p) has the midpoint property (Proposition 13.31) . all ball-convex sets in B are 
mid-convex and we obtain 

Corollary 3.9. The metric space {B,p) has normal structure. 

By Corollaries 13.61 13. 9[ and Theorem 12.21 we immediately get 

Theorem 3.10. Let dimK < oo. // a group G of p-isometric maps of B has at least one 
orbit separated from the boundary, then it has a fixed point. 

Since biholomorphic mappings of B are isometric with respect to the Caratheodory metric, 
we get also 

Corollary 3.11. Let H and K be complex, dimi^ < oo. // a group G of biholomorphic 
automorphisms of B has at least one orbit separated from the boundary, then it has a fixed 
point. 
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4 Fixed points for commuting holomorphic maps on 
the operator ball 



We continue to suppose that dimi^ < oo. The spaces can be real or complex. 

Lemma 4.1. Let G be a commutative semigroup of p-nonexpansive maps of B. If it has a 
p-bounded orbit, then it has an invariant m-set. 

Proof. Let X E Bhean operator whose orbit G{X) is p-bounded. We introduce the following 
partial order on G: g >~ h if and only if g = hf for some f E G. The semigroup G is a 
directed set with respect to this order (we use the standard definitions from [T, Chapter I]). 

Let r = diam(G'(X)) and let F = {Y e B : limsupg^^G,^) pC^^9{^)) < r}- Then 

^ = n U n ^h(X),r+e- 
£>0 g€G hyg 

Since each p-ball Ez,p is convex in the usual sense (see Lemma [3.5( 1)) and is mid-convex, 
the same is true for F. Also it is easy to see that F is p-bounded and is closed in the 
topology defined by the metric p. By Lemma [3.41 F is norm-closed. By Lemma [3.5( ii). F is 
WOT-compact, thus F is an m-set. 

Let us check that F is G-invariant. Indeed, if Y E F and h & G, then 

limsupp(/i(r),c/(X)) = limsupp(/i(F),/i/(X)) < limsupp(r, /(X)) < r. 

9e(G,^) fe{G,)~) fe{G,y) 

This means that h{Y) G -F. □ 

The idea of our proof of Lemma 14.11 goes back to [H] . 

Lemma 4.2. // a p-nonexpansive map g : B ^ B has an invariant bounded set U , then it 
has a fixed point in B. 

Moreover, if E is an arbitrary invariant m-set containing U , then a fixed point can be 
found in E. 

Proof. The assumption is equivalent to the condition that the semigroup generated by g 
has a bounded orbit. By Lemma 14. H there is an invariant m-set for g. Since m-sets are 
WOT-compact, there exists a minimal invariant m-set F for g. The m-span Fi of g{F) is 
contained in F and is invariant (indeed g{Fi) C g{F) C Fi). Hence Fi = F. If F is a 
singleton, then it is a fixed point. 

Now assume that F is not a singleton. By Theorem 13. 8[ there is r < diam(F) such that 
the set 

F{r) ■.= {Y eF : \/Z E F p{Y, Z) < r} 

is non-empty. Since F{r) is the intersection of F with a family of balls, it is an m-set. 
Moreover F{r) is invariant. Indeed, if F G F{r), then p{g(Y)^ g{X)) < r for all X E F. 
Hence the ball Eg(Y),r contains g{F). Since F is the m-span of g{F), this ball contains F. 
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Hence giY) E F{r). It is also clear that F(r) is a proper subset of F. We get a contradiction 
with the minimality of F. 

The second statement of the lemma follows from the construction. □ 

Using Lemma 1421 we prove the following analogue of Shafrir's theorem [2^ Theorem 3.2] 
for B: 

Theorem 4.3. A finite set S of commuting p-nonexpansive maps of B has a fixed point in 
B if and only if S has an invariant set separated from the boundary ofB. 

Proof. The "only if" part is trivial, we prove the "if" part only. Recall that a set is separated 
from the boundary if and only if it is p-bounded. If a set £^ = {gi, ...,(?„} has a p-bounded 
invariant set, then the semigroup generated by £ has a bounded orbit. By Lemma [4. 11 there 
is an invariant m-set for £. Let F be a minimal invariant m-set for £. 

Let g = gig2...gn- By Lemma [4.21 g has a fixed point in F. Let R be the set of all fixed 
points of g in F. 

We claim that gi{R) = R for each i. The inclusion gi{R) C R follows from commutativity. 
On the other hand, if X G -R, then X = g{X) = gi(Y), where Y = h{X) and h is the product 
of all gj with j ^ i. Clearly Y e R, X e gi{R), thus, R C gi{R). 

Let K be the smallest m-set containing R, clearly K G F. If i^' is a singleton, then 
R = K contains only one point which is a common fixed point for (^i, ... , 

If K is not a singleton, then, by Theorem 13. 8[ there is r < diam(i^) such that the set 
Kr = {X G K : p{X, Y) < r, for all Y G K} is non-empty. Since Kr is an intersection of an 
m-set with a family of p-balls, it is also an m-set. 

Let us show that Kr is invariant under all gi, i = 1, . . . ,n. First we note that Kr = {X G 
K : p(X,y) < r, for all Y G R}. Indeed, if p(X, F) < r for all F G i?, then R C Ex,r 
whence K C Ex.r because Ex^r is an m-set. Now if X G Kr, Y G R, using the fact that 
giiR) = R, we obtain p{gi{X)^Y) = p{gi{X), gi{Y')) < p(X, F') < r. By minimality of F, 
we get Kr = F. We get a contradiction since K G F and Kr is a proper subset of K. □ 

Note that Theorem 14.31 extends to infinite families of p-nonexpansive maps if these maps 
are assumed to be WOT-continuous (indeed in this case the set of all fixed points for each 
finite family of maps is WOT-compact). 

Since holomorphic maps are p-nonexpansive. Theorem 14.31 has the following corollary. 

Corollary 4.4. Let K and H be complex spaces. A finite set £ of commuting holomorphic 
maps of B has a common fixed point in B if and only if £ has an invariant set separated 
from the boundary of B. 
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5 Operator-theoretic applications: the case of real spa- 
ces 

For applications to invariant subspaces and orthogonalization, we need to prove that p is 
invariant under maps on B which are induced by J-unitary operators of the space Ti = H®K. 
In the case of complex scalars this follows immediately from the fact that these maps are 
holomorphic, while in the real case we need some additional arguments. 

We remind some definitions and constructions from the theory of spaces with an indefinite 
product. We refer to [1] or [T3] for more information. 

An operator U ojiTiis called J-unitary if it preserves the indefinite scalar product 

[x, y] = {Phx, y) - {Pkx, y), (5.1) 

where Ph and Pk are the projections on the summands in the decomposition Ti = H ® K . 
To each J-unitary operator U onH there corresponds a map wu of B in such a way that 

wuiU2 = °U!U2- (5.2) 

To see this recall that a vector a; G ?i is called positive [neutral, negative) if [x,x] > 
= 0, [x,x] < 0, respectively). A subspace of 7i is called positive [neutral, negative) 
if all its non-zero elements are positive (neutral, negative, respectively). For each operator 
X G S the set 

S[X) = {Xx®x: xeK} 

is a negative subspace of Ti. Since dim(S'(X)) = dim[K), S[X) is a maximal negative 
subspace in Ti. Indeed, if some subspace M oiTi strictly contains S[X), then its dimension 
is greater than the codimension of iJ, whence M (1 H ^ {0}. But all non-zero vectors in H 
are positive. 

Conversely, each maximal negative subspace Q oiTi coincides with S[X), for some X E B. 
Indeed, since Q H H = {0}, there is an operator X : K H such that each vector of Q is 
of the form Xx © x. Since Q is negative, we have [Xx © x,Xx © x] = ||Xa;|p — < 0. 
Since K is finite dimensional, this implies ||X|| < 1, so X G i3. Thus Q C 5'(X); and, by 
maximality, Q = S[X). 

It is easy to see that the map X S[X) from B to the set S of all maximal negative 
subspaces is injective and therefore bijective. 

Now we can define wu. Note that if a subspace L of ?^ is maximal negative, then its 
image UL under U is also maximal negative (because U is invertible and preserves [■,■]). 
Hence, for each X e B, there is Y e B such that S[Y) = US[X). We let wu[X) = Y. 

The equality (15. 2p follows easily because 

S[wuAwu2{X))) = UiS[wu2[X)) = UiU2S[X) = S[wu,u2{X)) 
and the map X — > ^(X) is injective. 
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Let U = {Uij)1j^^ be the matrix of U with respect to the decomposition H = H Q) K. 
Then U{Xx ® x) = {UuXx + Uux) © {U21XX + U22X). Since U{Xx © x) G S{wu{X)), we 
conclude that 

wuiX){U2iXx + U22X) = UiiXx + U12X. 

Thus 

wu{X) = (f/nX + f/i2)(f/2iX + U22)-'. (5.3) 

Let us denote by T the group of all such transformations of B (Helton [8j calls them general 
symplectic transformations) . It should be noted that all Mobius maps belong to T. Namely 
Ma = Wu where U is the J-unitary operator with the matrix {Uij)fj^i, where 

Uu = (Ih - AAT'^', Ui2 = -A{1k - A*A)-'/^, 
U21 = -A*{1h - AA*)-'^^, U22 = {Ik - A*A)-^'\ 

If a map if has the property V5(0) = 0, then it can be written in the form ^{X) = V1XV2 
where Vi, V2 are unitary operators in H and K, respectively. Indeed, let U he a. J-unitary 
operator with (p = wu, then UK = US{0) = S{ip{0)) = S{0) = K. Since if is a [■,■]- 
orthogonal complement of K in 7i, it is also invariant under U . Thus U has a diagonal 
matrix with respect to the decomposition Ti = H ® K: Uu = 0, U21 = 0. A moment 
reflection shows that Uu and U22 are unitary operators in H and K, respectively. Thus 
^(X) = U11XU22 is of the needed form. 

It follows now that \\^{X)\\ = \\X\\ for all X E B, ii ip{0) = 0. 

If (fi E T sends A to 0, then the transformation ipi = (f o Ma preserves and sends 
M_a{B) to ip{B). Thus, by the previous statement, ||v?(5)|| = ||M„a(5)|| and p(A, S) = 
tanh"^ yiB)\\ = p{0,ip{B)) for each E T with ip{A) = 0. 

This implies that p{{p{A), (p{B)) = p{A, B) for all A,BeB and ip eT . Indeed, let a G T 
with a{A) = 0, then p{A, B) = p(0, a{B)). Setting = a o Lp-^ we get that ^i^p{A)) = so 

p(^(A), ifiB)) = p(0, H^m) = P(0, aiB)) = piA, B). 

Thus T is a group of isometrics of {B, p) . Applying Theorem 13.101 we obtain 

Corollary 5.1. // a group G of general symplectic transformations of B has an orbit sepa- 
rated from the boundary of B, then it has a fixed point in B. 

Recall that subspaces M, X of a space with indefinite scalar product TC form a dual pair 
of subspaces if M is positive, N is negative and M + N = Ti. 

A Pontryagin space Ilk is the space 7i = H ® K with the scalar product (15.11) . where H 
and K are Hilbert spaces, dim if = 00, dimi^ = k. 

Corollary 5.2. Any bounded group of J-unitary operators in a real Pontryagin space Ilk 
has an invariant dual pair of subspaces. 



14 



Proof. Let F be a bounded group of J-unitary operators in H, C = sup^/gp ||?7|| and G = 
{wu : U G r}. Then the orbit G(0) is separated from the boundary in B. Indeed, let 
X = wu{0), for some U eT. For each x E K, the vector Xx © x can be written as ?7(0 © y) 
where y e K. Hence \\Xx © < C^||2/|P and [Xx ® x,Xx ® x] = [0 © © y]. Thus 
— = — and ||Xa;|p + ||a;|p < C^||2/|p. It follows that < r^||x|p where 




^ < 1. Thus ||X|| < r for all X G G(0). 



By Corollary I5.H G has a fixed point X E B. Hence the maximal negative space = 
S{X) is invariant under F. Denoting by M the [■, -J-orthogonal complement of it is easy 
to see that (M, A^) is an invariant dual pair of subspaces. □ 

Now we turn to the problem of orthogonalization of a representation of a group on a 
Hilbert space (the standard reference of the topic is the book [H]). As for representations 
in complex Hilbert spaces (see we say that a representation vr of a group F in a real 

Hilbert space Ti preserves a sesquilinear form with finite number of negative squares if 7i 
can be decomposed into the orthogonal sums of subspaces H and K (with dimi^' < oo) in 
such a way that {PH7r{g)x, 7r{g)y) - {PKn{g)x, n{g)y) = {PhX, y) - {PrX, y) for all x,y eH 
and all g E G. 

Corollary 5.3. If a bounded representation of a group F by operators in a real Hilbert space 
Ti preserves a sesquilinear form with finite number of negative squares, then it is similar to 
an orthogonal representation ofV on Ti. 

Proof. By our assumptions, tt acts by J-unitary operators on a H^ space H ® K. Since 
Fo = 7i"(F) is a bounded group of J-unitary operators, the previous Corollary shows that 
there is an invariant for Fq dual pair (M, A^) of subspaces. Moreover A^ = UK for some 
J-unitary operator U . Then all operators T{g) = U~^7i{g)U are J-unitary, and the subspace 
K is invariant for them. It follows that H is also invariant for operators T{g). Hence these 
operators preserve the scalar product on H. Thus g i-^ T{g) is a unitary representation 
similar to vr. □ 

Acknowledgement. The authors are grateful to Professor S. Reich for helpful sugges- 
tions and references. 
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